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Executive Summary 



A major requirement for states under the No Child Left Behind Act of 2001 (NCLB) was to plan and 
implement a statewide accountability system that met highly specific criteria. Each state submitted its initial 
draft accountability plan, called an “educational accountability workbook,” to the U.S. Department of 
Education (ED) in January 2003. Following a peer review process and responses from ED, all states re- 
submitted their accountability plans with required revisions and received approval for these plans by June 
2003. Shortly thereafter, the Council of Chief State School Officers (CCSSO) prepared and released an 
initial report that examined common issues and themes in states’ accountability workbooks and ED’s ensuing 
approval decisions (Erpenbach, Forte Fast, & Potts, 2003). 

The purpose of the present paper, is to summarize the predominant issues presented by states in their 
requests to amend their state accountability plans to date. By late summer 2003, some states had already 
begun submitting amendments to their accountability workbooks; the pace of submissions accelerated 
through the spring and summer of 2004. A number of factors may have contributed to this acceleration, 
including the high stakes consequences of the NCLB accountability requirements, the additional federal 
flexibility allowed for several provisions of the requirements, delays in the provision of regulations and 
guidance from ED, exchange of information among states on elements of their plans, and the initial release 
of 2002-03 AYP results. With 47 states submitting amendment requests by the April 1 , 2004 deadline set by 
ED, the Accountability Systems and Reporting State Collaborative (ASR-SCASS) and CCSSO recognized 
the need for a paper to summarize the core issues states were trying to address with the amendments. 

Information supporting this summary and analysis was provided to CCSSO by state education officials who 
voluntarily forwarded their amendment requests and approval letters from ED. Some information was also 
obtained from websites, with follow-up calls to staff of state departments of education. To protect states’ 
privacy, states are only named in this summary when the state request was described in a public official 
response letter from ED; other descriptions of state requests are kept “state blind.” 

This paper is intended to summarize areas of the NCLB legislation’s accountability requirements where 
states have sought substantive changes to their accountability plans or have sought targeted technical 
assistance. Complete information on state accountability systems is reported by states on the CCSSO web 
site through 50-state accountability profiles (http://accountability.ccsso.org/). The website includes up-to- 
date information on state accountability systems and incoiporates state accountability plan amendments for 
2003-04. 

To facilitate interpretation, this paper generally follows the same structure as its predecessor (Erpenbach, 
Forte Fast, & Potts, 2003). The content of this report is organized as follows: 

Section I presents an introduction and develops the background related to this paper. 

Section II describes the central issues addressed in the amendments proposed by states. 
Section III offers a set of conclusions reached as a result of our review of states’ accountability 
workbook amendments and ED’s responses. Also identified are “unanticipated approvals” and 
amendment requests awaiting ED decisions. 

Appendix A provides a summary of ED’s State Accountability Workbook Decisions that Changed 
Between 2002 and 2004. 

Appendix B explains the 1 0 principles states were asked by ED to describe in their accountability 
plans. 

Neither CCSSO nor the authors necessarily endorse any of the educational accountability strategies embedded 
in the amendments described in this paper. The purpose of this paper is to describe amendment proposals, not 
to judge any state’s intent in making specific proposals or whether these proposals are appropriate or technically 
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sound. Neither is this paper intended to serve as an evaluation of ED’s responses or failure to respond as may be 
the case. No recommendation of sound, appropriate, or effective practice is implied or should be inferred. 

Summary of Accountability Amendment Requests from States To Date 

This report organizes into four main categories the requests for amendments to accountability systems and 
plans submitted by states to ED by the April 1, 2004, submission deadline. The categories were created for 
the purpose of summary and discussion in this paper. The outcomes of these requests are discussed in the 
body of the paper. 

1) Standards and Assessments: One-fourth of states submitted amendment requests related to their 
standards and assessments, including the following: 

Changing the test or grade level that yields scores used in AYP analyses; 

“Banking” of test results for use in AYP determinations; 

Using of out-of-level tests or other alternate assessments; 

Adjusting assessment scores for defining “proficient.” 

2) AYP Model: Almost all states that submitted amendments requested at least one change to their 
AYP model. These requests can be categorized as follows: 

AYP application rules including requiring two years of missing targets for identifying schools and 
districts, retroactive application, and defining Title I students; 

AYP indicators including the use of rounding rules and defining percent proficient on required 
assessments, student participation rate in assessments, and other academic indicators such at the 
graduation rate; 

State targets and schedules for AYP including establishment of annual measurable objectives and 
intermediate goals; 

Strategies to enhance reliability of AYP results, including establishing minimum “n” sizes, using 
confidence intervals, and applying safe harbor reviews. 

3) Inclusion of All Students in Accountability: Almost all states requested amendments to take 
advantage of the flexibility issued by ED regarding the following: 

Limited English Proficient (LEP) students’ assessment and inclusion in accountability systems; 
Students with disabilities’ inclusion in alternate assessments and accountability determinations. 

4) AYP Consequences and Reporting: A small number of states requested amendments in this area, 
such as the following: 

Timing of consequences for schools; 

Implementing state rewards and recognition; 

Organizing for reporting, including integration with state reports. 
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Requests Consistently Not Approved 

To date, ED has been consistent in not approving several categories of requested changes from states. ED 
responses regarding the items on the following list have been consistent. These items can be considered 
“non-negotiable” types of requests, as they have been consistently denied: 

Using out-of-level testing results in AYP calculations, except when such practice is consistent 
with the 1 percent cap requirements; 

Limiting identification of schools as in need of improvement to those schools that miss AYP for 
two consecutive years in the same content area and student subgroup; 

Applying retroactively any 2003-04 amendments to years prior to the 2003-04 school year; 
Exempting schools from the obligation to test students with significant medical-emergencies via a 
blanket policy (decisions must be made on a case-by-case basis); 

Switching the order of the NCLB-prescribed sanctions of school choice and supplemental 
educational services — school choice must be offered in the first year of identification. Further, 
school choice cannot be limited to “non-proficient” students. 

In the following sections of the paper, the authors set the context for the discussion of state amendments to 
their accountability systems, describe specifics about the approach and rationale for selected examples of 
state requests that were approved, and provide the rationale given by ED for requests not approved (where 
that rationale is known). 
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Section I: Introduction and Background 

Under the No Child Left Behind Act of 2001 (NCLB), all U.S. states, the District of Columbia, and Puerto 
Rico were required to submit State Educational Accountability Workbooks (workbooks) to the U.S. 
Department of Education (ED) by January 31, 2003 1 . Each workbook was organized using 10 categories, 
as established by ED. (Appendix C includes a list of these 10 principles.) For example, all states were 
required to present a plan for a statewide accountability system that included all public schools (K-12) and 
included all students enrolled in a school. For many states that had already established statewide 
accountability systems based on state policies, the transition to the specific NCLB requirements involved 
adaptation of existing systems and often change in guiding assumptions and methodology. For almost all 
states the NCLB requirements involved expansion and development of assessments, data system capacity, 
and methods for reporting and dissemination of results. 

The initial workbooks comprising state plans presented by states were reviewed by peer teams and by ED 
staff in spring 2003. The general process following these reviews involved a series of negotiations and 
interactions conducted between state leaders and federal officials. All 52 revised workbooks were 
tentatively approved by June 2003. In most cases, ED required states to make some changes to their 
workbooks prior to granting approval. In all cases, approval remained contingent on implementation of the 
plans as described in the workbooks. (See ED language included in direct correspondence to states 
regarding their accountability plans, available at http://www.ed.gov/admins/lead/account/letters/ 
index.html.) 

Even before gaining this initial approval for their plans, states had to begin implementing many aspects of 
their plans in order to meet the requirement for making and publicizing their adequate yearly progress (AYP) 
decisions prior to the start of the 2003-04 school year. For example, states had to communicate their 
accountability plans to their stakeholders; adopt protocols for preparing their assessment, graduation, and 
other academic indicator data for AYP analyses; develop reporting mechanisms that met the NCLB 
requirements; develop plans to support students taking advantage of mandated school choice options; and 
develop processes and staffing for handling anticipated appeals to AYP determinations. Most states invested 
substantial staff time to these tasks during the spring and summer of 2003. After this period, many shifted a 
portion of their investments to the study of that first NCLB AYP cycle: examining what worked and what 
required refinement. This led most to think about modifications to their accountability systems; a process 
that was further stimulated by the issuance of regulatory changes in winter 2003-04. 

States that make “significant changes” to their plans under Title I — including their assessment and 
accountability systems — may request federal review of those changes at any time (sec. 1 1 1 1(f)(2)). Toward 
this end, some states began submitting proposed amendments as early as summer 2003. Anticipating that 
many states intended to amend their educational accountability systems and desiring to streamline the 
process, the newly appointed Assistant Secretary for Elementary and Secondary Education, Raymond Simon, 
sent a letter on February 5, 2004, to chief state school officers outlining procedures for how states could 
request federal approval for amendments to their accountability plans. This letter set a deadline of April 1, 
2004, for submission of amendments that states intended to apply to their 2003-04 data in order to make 
2004-05 AYP decisions. Many states took advantage of this opportunity and Simon pledged that ED would 
“make every effort” to respond to any requests within 30 days 2 . These requests for accountability plan 
amendments and the ensuing federal responses 3 are analyzed in this paper. 

1 For more complete information on the initial submission and review process, see the recent CCSSO report (Erpenbach, 
Forte Fast, & Potts, 2003), available from CCSSO publications at http://www.ccsso.org/publications/ 

details. cfm?PublicationID=2 1 5. 

2 According to an Education Week article, “ . . . while department officials have been in constant contact with states, by 
late last week they had given final, written approval to only 20 of the approximately 45 that submitted requests by April 
1” (Olson, 2004, July 14). By August 2004, ED had responded to many more of the plan amendment requests. 

3 Response letters to the states are available at http://www.ed.gov/policy/elsec/guid/stateletters/index.html. 
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The present paper summarizes states’ requests for amendments to their educational accountability plans and 
does not represent a comprehensive statement of the current status of these plans, nationwide. Many states 
requested changes that were already part of other states’ approved plans, so the amendment requests do not 
necessarily reflect the prevalence of any specific strategy. And, although nearly all of the 50 states, Puerto 
Rico, and the District of Columbia submitted multiple amendment requests sometime in the past year, a few 
have not submitted any requests at all. 

A number of factors may have contributed to the large number of amendment requests in 2003-04, such as 
the high stakes consequences of the NCLB accountability requirements, recently announced federal 
flexibility regarding several provisions of the requirements, delays in the provision of regulations and 
guidance from ED, exchange of information among states on elements of their plans, and the initial release 
of 2002-03 AYP results along with projections for subsequent years. Decisions resulting from the 2003-04 
round of accountability plan amendments are also likely to trigger additional amendment requests next year 
or in following years. 

As noted above, a few states did not submit amendments to their plans as they now stand. These states may 
be waiting for political events to unfold or for decisions within their own jurisdictions, conducting additional 
analyses to support their request, or have local factors related to their plan or planned changes that prevented 
a request this year. 

Finally, this paper addresses publicly available state requests for accountability amendments. Some states 
may have requested amendments without making these requests public; thus, it is possible that a number of 
requests have been made that could not be captured here. 



Timeline of Significant Events Related to State Accountability Plans 

The NCLB Act Enacted (January 2002) 

Standards & Assessment Regulations Issued (July 2002) 

Accountability Regulations Issued (early December 2002) 

State Meetings with ED Officials Begin (December 2002) 

ED Releases Accountability Workbooks (late December 2002) 

Draft State Accountability Workbooks Due to ED (January 31, 2003) 

Peer Reviews of State Accountability Plans (January through April 2003) 

All State Accountability Plans Tentatively Approved (by early June 2003) 

CCSSO Publishes Paper on State Plans and Approval Decisions (July 2003) 

States Begin Submitting Plan Amendments to ED (Fall 2003-spring 04) 

Regulations Regarding Achievement of Severely Cognitively Disabled Students Issued 
(December 9, 2003) 

Simon Letter to Chiefs Setting April 1 for Plan Amendments (February 2, 2004) 

Secretary's Letter to Chiefs Announcing New Flexibility Related to LEP Students (February 
20, 2004) 

Secretary's Letter to Chiefs Announcing New Flexibility Related to SWDs (March 2, 2004) 
State Amendments Due to ED for Decisions Applied in 2003-04 AYP Determinations (April 
1, 2004) 

ED Response Letters Begin (May 2004) 
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Section II: Key Issues in States' Amendments to 
Their NCLB Accountability Plans in 2003-04 

The central issues addressed in states’ educational accountability workbook amendments and described in 
Part II of this document are organized into the following major categories: 

Standards and Assessments; 

Adequate Yearly Progress (AYP) Model; 

Inclusion of All Students in Accountability; 

AYP Consequences and Reporting. 

Standards and Assessments 

About one quarter of the states seeking amendments to their NCLB accountability plans submitted requests 
related to their standards and assessments. These requests fall into the following categories (some states 
requested amendments in more than one category): 

Changes to the test or grade level that yields scores used in AYP analyses, as indicated in the 
original plan; 

“Banking” of test results for use in AYP determinations when students have multiple 
opportunities to take the test used for AYP purposes; 

Use of out-of-level tests (also termed instructional level tests) or other alternate assessments for 
students with disabilities or students with limited English proficiency; and 
Adjustment of the scores students must achieve to be considered proficient. 

When interpreting the information in this section of this paper, it is important to note that all states, the 
District of Columbia, and Puerto Rico will be required to submit evidence pertaining to their standards and 
assessments by the spring of 2006 for a separate federal review. This review will encompass new standards 
and assessments as well as standards and assessments that were reviewed under the Improving America’s 
Schools Act of 1994 if their results will be used in the NCLB-related AYP determinations. Some states may 
be waiting to request accountability amendments until the completion of this next round of standards and 
assessments reviews. 



Changes to the Test or Grade Level Used for AYP 

Most states will be making significant changes to their assessment systems over the next few years, but, so 
far, only a few have addressed these in their requests for amendments. In some cases, states simply indicated 
a change in the test that will yield the scores used for AYP purposes or to meet other NCLB requirements. 

For example, Alabama was approved for a switch to the English Language Development Assessment 4 
(ELDA) for measuring English proficiency. Another state requested a similar switch for one of its academic 
assessments. 

Tennessee, which already tests in grades 3-8 and originally planned to use assessment results for all of these 
grades in AYP analyses, asked to use results from only a subset of these grades for the years prior to 2005-06 
(the year in which assessment in each of these grades is required by NCLB). Some arguments for this request 
pointed to a misalignment between uses of the data for AYP and the uses for which the state had developed 
its current assessments. Fairness was also at issue: states that already test in these grades may need the same 
amount of time as other states (i.e., until the 2005-06 school year) to prepare for the full testing requirement. 
As of August 2004, ED had not yet provided a response to these requests. 



4 See http://www.ccsso.org/projects/SCASS/Projects/Assessing_Limited_English_Proficient_Students. 
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Similarly, other states indicated that a set of tests or the policies associated with their implementation would 
not take effect prior to 2005-06 as had been anticipated. Therefore, the source of the assessment data used for 
AYP determinations would be somewhat different than originally described for 2003-04 and 2004-05. In fall 
2003, Maine proposed the use of results from local assessments for AYP purposes. However, the state later 
withdrew this request after it determined that the federal assessment and accountability requirements might 
prove overwhelmingly burdensome and intrusive for its instructionally-focused local assessment system. 
Maine will continue its support for the local assessments but will also develop statewide assessments for use 
in AYP measures. 

Banking of Test Results 

A number of states received approval for the practice of “banking” test results when students have two or 
more opportunities to take a test that yields scores used in AYP analyses. The version of this practice that has 
evolved into what is now acceptable to ED involves the designation of a single test administration as the 
“official” point at which students are expected to have acquired the knowledge and skills specified in the 
standards on which the test is based. A state may allow students multiple opportunities to take the test before 
this point. That is, if the test is based on standards relevant to the end of 1 1 th grade, test administrations 
before this time may be considered “early” and the administration in the spring of 11 th grade “official.” If a 
student takes the test before the official administration and achieves an acceptable score (i.e., proficient), that 
score is “put into the bank.” If that student scores below the acceptable level, the score is not banked and the 
student must take the test again. The re-testing student may continue to test as often the state allows, but the 
scores will not count for AYP judgments until he or she passes or until the official administration, whichever 
occurs first. States using this strategy more often do so at the high school level when the test is a requirement 
for graduation. States may ha nk for participation rate determination just as for performance: only passing 
attempts count prior to the official administration. 

Oregon’s original accountability plan included a form of score banking that allowed credit for performance 
on computer-adaptive tests administered on-demand during specified testing windows each year, including 
multiple opportunities to re-test when the passing standard is not met. ED rejected this strategy during the 
initial reviews in 2003, citing the NCLB regulation (§200. 20(c)(3)) requiring states to use the first score a 
student obtain s in their AYP calculations and the lack of a clear way of calculating a participation rate. The 
“first score” requirement described in §200.20(c)(3) had no basis under NCLB and ED later reversed its 
position (Simon, January 26, 2004) in response to Oregon’s December, 2003, request for amendment. Thus, 
Oregon is now allowed to implement its original, previously-denied banking strategy and will use the spring 
administration as the “official” one for AYP purposes. 

In spring 2004, several other states requested amendments to apply banking rules. At least one would like 
districts to choose when students have the opportunity to take a test, as long as it is administered by the end 
of 1 1 th grade. Another state has asked for approval of its plan to use results from re-testing opportunities that 
occur after the official administration of a test; the re-testing students would be administered new test items 
covering only the standards where he or she had not previously met the passing criteria. 

One other state has requested permission to use results from all grades eligible to take the test rather than 
results from only the official administration in AYP calculations for its districts. This state administers a test 
based on high school standards that students may begin attempting while in middle school. The state had 
initially planned to start using the results from only the highest grade level at which the test could be taken, 
then use results from the highest two grade levels in the following year, the highest three grade levels the 
following year, and so on until results from all eligible grade levels would be included. The state would like 
to amend its plan so that results from all grade levels are included in the district and state AYP analyses each 
year beginning immediately; for school-level AYP, only scores for high school students would be counted. It 
is not clear from the documentation how the state intends to calculate participation rate. 
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Out-of-Level Tests or Other Alternate Assessments 

Out-of-level testing involves the administration of a test form based on one grade level’s content 
expectations to a student who is enrolled at another, usually higher, grade level. Under regulations 
promulgated on December 9, 2003, and consistent with Secretary Paige’s March 2, 2004 letter, out-of-level 
testing is effectively now defined by ED as a test based on alternate academic content and student 
achievement standards for severely cognitively disabled students and subject to the law’s 1 percent cap 
restrictions for considering student performance as proficient in AYP determinations. Under these conditions, 
such tests may be used as part of a state’s assessment and educational accountability systems. 

During the initial review and approval cycle in early 2003, no state was allowed to use results from out-of- 
level testing in AYP calculations 5 ; however, this position changed over time and by June 2003, ED was 
approving the use of out-of-level assessments for students with disabilities (SWDs) to make AYP 
determinations within a narrow range of parameters (see Erpenbach, Forte Fast, & Potts, 2003). In these 
instances, ED referred to out-of-level assessments as “instructional level assessments” (Paige, June 27, 2003). 
The use of out-of-level testing continued to be prohibited except in cases where it was argued that out-of-level 
results are considered necessary for instructional purposes and the student also takes an on-grade test for AYP 
purposes. Prior to this time, states had not been allowed to use the results of out-of-level testing in AYP 
calculations because these tests were, by definition, not aligned to on-grade expectations. 

One state requested an amendment allowing out-of-level tests to be offered as an option for severely 
cognitively disabled students. Four others requested approval for the use of out-of-level tests as one of the 
options available for including students with non-severe disabilities in the assessment system. ED responded 
in one of these cases shortly after issuing the final regulations on the inclusion of students with the most 
severe cognitive disabilities in Title I assessment (Federal Register, December 9, 2003), specifically rejecting 
the use of proficient scores from out-of-level tests in AYP calculations unless the scores are included in a 1 
percent cap: 

The Department has issued a final regulation that permits a State to use alternate 
achievement standards to evaluate the performance of students with the most significant 
cognitive disabilities, and to give equal weight to proficient achievement based on the 
alternate standards in calculating school, district, and State AYP, provided that the number of 
proficient scores based on the alternate achievement standards does not exceed 1.0 percent 
of all students in the grades tested at the State or LEA level. Any use of out-of-level 
assessments for AYP purposes would be subject to this regulation. The final rule does not 
alter the IEP team’s role in making individual decisions about how to assess a child. Instead, 
it restricts, solely for purposes of calculating AYP, the number of scores that can be counted 
as proficient based on alternate achievement standards. Achievement standards associated 
with out-of-level assessments may meet the requirements of this regulation only if they are 
aligned with the State’s academic content standards, promote access to the general 
curriculum, and reflect professional judgment of the highest achievement standards possible. 

The results from out-of-level assessments must be included within the 1.0 percent cap for the 
purposes of calculating AYP, because the achievement standards associated with the content 
measured by out-of-level assessments are clearly different from the achievement standards in 
the target grade (Tomalis, December 31, 2003). 



5 In promulgating regulations for NCLB standards and assessment provisions, ED stated, “Although out-of-level tests, 
for example, may provide instructional information about a student’s progress, they are not an acceptable means to meet 
the State’s assessment requirements under §§200.2 and 200.6 or the accountability requirements of the NCLB Act.” 

( Federal Register, July 5, 2002, pp. 45044-45045). 
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None of the other three states that have publicly requested permission to continue using out-of-level tests has 
received a public response from ED; however, sources in at least one of these states suggest that ED may 
passively allow the use of out-of-level tests for AYP puiposes simply by “looking the other way” until this 
state undergoes the Peer Review of its standards and assessments sometime in the next two years. This state, 
and at least one of the others, cites the need for more time to develop and implement other alternate 
assessments for students with non-severe disabilities. 

In early 2004, West Virginia proposed the use of “an equivalent” form of its statewide academic assessment, 
WESTEST, that would be aligned to grade-level standards but would be more accessible for students with 
cognitive impairments. The state does not intend for the results from this assessment to be included in the 1 
percent cap because it is to be based on grade-level expectations; therefore, there would be no limit on the 
number of students who could score at or above the proficient level and have their scores included in AYP 
determinations (provided that the assessment would meet all of the relevant NCLB Act requirements). 
Although approval of the assessment itself could occur only as part of the upcoming standards and 
assessment reviews, ED did respond to this basic request signaling a willingness to consider the state’s 
proposal: 

This approach would be in alignment with the statute and regulations if the ‘equivalent’ assessment 
is aligned to grade-level achievement standards and content standards. In such an instance, there 
would be no limit on the number of proficient scores that could be included in AYP assessments 
(Simon, March 4, 2004). 

Several other states have made requests to use the results of new alternate assessments for severely 
cognitively impaired students in their AYP calculations. Those that have received approval to date include 
Alabama, Indiana, and Nebraska. Maine has specifically been asked to ensure that the results from its new 
alternate assessment are included in its AYP calculations, although it is not clear that the state ever asked to 
do otherwise. (See the “ Inclusion ” section of this paper for additional information on these issues.) 

Adjustment of Scores Designating Proficiency (“Cut Scores”) 

NCLB requires states to develop student academic achievement standards (previously called performance 
standards) that define at least three levels of performance. These levels must include one that corresponds to 
grade-level proficiency (proficient), at least one level above proficiency (advanced), and at least one level 
below proficiency (basic). As part of the development process, states must identify the test scores that 
differentiate between these levels, referred to as “cut scores.” 

Although the definition of “proficient,” including the score a student must achieve on a test in order to be 
classified as proficient, has much to do with the meaning of AYP, this has been the subject of only three 
amendment requests to date. Two states requested amendments reflecting changes in their student academic 
achievement standards due to the introduction of a new assessment. One other state made an adjustment in 
the cut score separating its proficient level from the level below it on an existing test. This adjustment was 
described as the correction of an error that had been made during the original standards-setting process. So 
far, no state has requested an amendment indicating plans to revisit its student academic achievement 
standards in general. 

Adequate Yearly Progress (AYP) Model 

A state’s AYP model defines how it actually calculates adequate yearly progress for schools and districts. 
This model must address questions such as, “Which students are included in the denominator when you 
calculate the participation rate?” and “How does the state address the reliability of its AYP decisions?” 
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Over half the states that submitted amendments requested some change to their AYP model. These requests 
may be categorized as follows: 

AYP Application; 

AYP Indicators; 

Annual Measurable Objectives, Intermediate Goals, and Trajectories; 

Strategies Intended to Enhance Reliability. 

AYP Application 

Several states requested permission to make changes in how and to whom AYP decision rules would be 
applied. Such requests related to how schools operating Targeted Assistance Title I programs calculate AYP, 
general triggers for identification, differential rules for identifying districts for improvement, and the 
retroactive application of approved amendments. 

Rules for Targeted Assistance Title I Schools 

For the first 20 years of its implementation, the Elementary and Secondary Education Act (ESEA) only 
allowed schools to operate Title I programs that targeted academic assistance to specific students 
(commonly referred to as “Targeted Assistance Programs”). With the Hawkins-Stafford amendments to 
ESEA in 1988, schools in which a significant proportion of the students come from low-socioeconomic 
backgrounds were given another option for providing Title I services. Instead of identifying specific 
students for service as part of a Targeted Assistance program (see sec. 1115 of Title I), these high-need 
schools could be granted permission to combine their Title I funds with dollars from other funding sources 
in order to implement Schoolwide services designed to improve the quality of education for all students in 
the school. Since these amendments were adopted, more and more schools have taken advantage of this 
flexibility in the use of their Title I funds, so that most high-need Title I schools now operate Schoolwide 
programs (see sec. 1114 of Title I) rather than Targeted Assistance programs. 

Under section 1116(b)(1)(D), AYP calculations for schools operating Targeted Assistance Title I programs 
may be based on the progress of all students in the school or on only students who are served — or are 
eligible to be served — under Title I s . No states approved in the initial round of accountability workbook 
submissions appear to have chosen the latter approach. 

North Carolina became one of the first states to amend its accountability plan so that some Local 
Educational Agencies (LEAs) — more commonly referred to as school districts — may chose to calculate 
AYP for schools that operate Targeted Assistance programs (as opposed to Schoolwide programs) based 
only on those students served or eligible to be served through Title I. ED approved this request but asked 
North Carolina to “conduct this analysis in alignment with ED guidance on this issue.” North Carolina 
will not allow districts to change a school’s program designation (e.g., from Schoolwide to Targeted 
Assistance) after the start of a school year. In addition, districts may choose to calculate AYP based on 
results for students who are served by Title I only if that group meets the state’s minimum “n” of 40. North 
Carolina subsequently confirmed with ED that states could exercise this approach every year and not just 
the year after which an AYP target has been missed. 

At least one other state has submitted a similar request but as of August 2004 had not received its response 
from ED. 



6 Title I funds are allocated to states, school districts, and schools on low income statistics; however, children eligible to 
be served under Title I programs are those a school identifies as “failing or most at risk of failing, to meet the State’s 
challenging student academic achievement standards” (sec. 1115(b)(1)). In other words, the students eligible to be 
served are not always those students who generate the funding. 
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Triggers for Identification 

At the most basic level, NCLB requires that schools or districts that miss their AYP targets in two 
consecutive years be identified for improvement. There are a number of possible permutations of this basic 
idea, and some states requested amendments related to the specific set of conditions that result in 
identification for improvement. Arkansas was approved (September 2003) to conduct comparisons of 
current-year assessment results with the average of the most recent three years’ results (including the current 
year results) and to use the most favorable data in making AYP determinations. ED required the state to (1) 
submit impact data on the number of schools and districts making AYP because this procedure was applied 
and (2) assure that the same procedures will be applied to each subgroup within a school. Washington was 
approved (June 18, 2004) to “move towards averaging data across grade levels, and will average data over the 
past two years for AYP determinations.” 

In early 2003, ED’s position was that a school or district missing any AYP measure for two consecutive years 
would result in identification for improvement. For example, a school in which a student group failed to meet 
the annual measurable objective (AMO) for mathematics in one year and a different group at the same school 
failed to meet its other academic indicator (OAI) the following year would be identified for improvement. 
That position evolved in subsequent approval decisions such that states may now restrict AYP determinations 
to the same subject (an amendment approved for North Carolina in May 2004 and Illinois in June 2004) or 
the same OAI (an amendment approved for Oregon in May 2004 together with its request for same subject 
determinations). ED, however, continues to steadfastly maintain that AYP determinations may not be 
restricted to the same subgroups within the subject or OAI; a subject that continues to be debated nationally 
given differing readings of the statutes and regulations among the states and other interested parties. 

ED’s rationale was included in a December 31, 2003, letter from Acting Assistant Secretary of Elementary 
and Secondary Education Ronald J. Tomalis to West Virginia State Superintendent of Public Instruction 
David Stewart in response to the state’s proposed change in how AYP would be determined (Utah was 
another state that sought to base the identification for improvement on the failure of the same subgroup in 
the same subject or OAI.): 

West Virginia proposes to identify schools and districts for improvement only if the same subgroup 
did not make AYP in the same subject or indicator for two consecutive years. This identification 
procedure is not consistent with the statute or regulations. As you know, there is flexibility to permit 
States to determine schools in need of improvement on the basis of not making AYP in the same 
subject for two consecutive years. This flexibility stems from other provisions in the statute that treat 
reading and mathematics independently — e.g., separate starting points and annual measurable 
objectives. These provisions recognize that student achievement in reading and mathematics in a 
State may be starting at very different points and thus the State would need to establish different 
trajectories to attain 100 percent proficiency. Concomitantly, it made sense to permit a State to 
identify schools in need of improvement based on not making AYP for two years in each subject. 
Subgroups, on the other hand, are not treated differently in the statute and thus the statute does not 
support similar flexibility to identify schools in need of improvement on the basis of “same 
subgroup” performance for two consecutive years. Moreover, your proposal is inconsistent with the 
statute’s accountability provisions, which require that each subgroup meet the State’s annual 
objectives in each subject each year. The intent of school identification is not to lay blame on a 
particular group of students, as the ‘same subgroup/same subject’ proposal would do, but to identify 
the instructional and academic elements that need to be improved. A school that is identified for 
improvement [the statute refers to identifying schools for improvement] should look to specific 
instructional remedies in the subject, other indicator, or participation rate. 



A New Approach to Identifying Districts for Improvement 

In one of the more surprising developments of the 2004 amendment process, at least eight states have 
received approvals to modify the method by which school districts missing AYP targets are identified for 
improvement. Prior to these amendments, all states calculated district AYP based on results for all students in 
the tested grades (or all students in the tested grades in Title I schools) in the district. Arkansas, Delaware, 
Iowa, Maine, North Carolina, Ohio, Tennessee, and West Virginia have now been granted permission to 
calculate AYP separately for up to three grade spans. That is, they can calculate AYP for elementary schools, 
AYP for middle schools, and AYP for high schools. For a district to be identified for improvement, it would 
have to miss AYP at each of these levels two years in a row. Further, these states can consider AYP separately 
for reading and mathematics such that a district would have to miss AYP in reading (for example) at all three 
levels two years in a row to be identified for improvement. ED’s response letters to Delaware, North 
Carolina, and Ohio noted that the state should “1) monitor districts that have not made AYP in one grade 
span but have not been identified for improvement to ensure they are making the necessary curricular and 
instructional changes to improve achievement, and 2) take steps to ensure supplemental [educational] 
services are available to eligible students from a variety of providers throughout the state (including in LEAs 
that have not been identified for improvement but that have schools that have been in improvement for more 
than one year).” 

The authors are aware of a number of other states that are preparing to request similar amendments and note 
that this strategy may result in many fewer districts being identified for improvement shortly before the 
federal elections in the fall of 2004. In a small state preparing to submit an amendment, for example, the 
number of school districts that would be identified for improvement based on 2003-04 determinations would 
drop from 100 to 15. In Tennessee, 92 percent of the school districts did not make AYP last year. The 
approved amendment dropped the proportion to 65 percent (Olson, July 14, 2004). 

Retroactive Application 

While most requests for amendments were specific to a particular component of a state’s accountability plan, 
some states submitted requests for more general flexibility. North Carolina requested retroactivity in the 
application of approved amendments to AYP calculations for the 2002-03 school year. Further, the state 
requested that in the event that ED did not approve such retroactivity, it be allowed to treat the 2002-03 
school year as a pilot year and the 2003-04 as the first year for official AYP reporting. Both of these requests 
were primarily based on North Carolina’s desire to ensure consistency and comparability of the 2002-03 
AYP decisions in the state with the 2003-04 decisions. ED denied both parts of this request: 

It would be inconsistent with Congressional intent, legal precedent, and the facts of implementation 
to allow the retroactive application of policies and decisions, even those made by a State. NCLB 
accountability requires that schools are only identified for improvement after not making AYP for 
two consecutive years. Schools that did not meet AYP last year because of state decisions about AYP, 
and then make AYP this year under new state rules, will not be identified for improvement. Thus, 
allowing for the retroactive application of policies and decisions made by States would result in no 
practical difference, and would also result in great confusion in the field about what rules applied 
when and how. To do otherwise would render state accountability systems ineffective and 
indefinable; consistency would be lost; and schools, teachers, parents and students would face a 
moving target that provided very little, if any, standard by which to measure progress and success 
(Simon, May 20, 2004). 

In another part of its response to North Carolina, ED made reference to a letter Secretary Paige sent to 
Senator Edward M. Kennedy on April 27, 2004 (Paige, April 27, 2004). The letter was in response to one 
Senator Kennedy 7 had written to Secretary Paige on March 24, 2004, requesting that states be allowed to 
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apply the “regulations” related to the inclusion of students with severe cognitive disabilities (. Federal Regis- 
ter, December 9, 2003) and students with limited English proficiency (February, 2004) retroactively to AYP 
calculations for the 2002-03 school year. Senator Kennedy noted that these regulations had not been released 
in a timely manner, suggesting that it would be appropriate to allow states to apply the new regulations 
consistently to NCLB AYP decisions for all years. In his response, Secretary Paige rejected the timeliness 
argument and highlighted the lack of precedent for retroactive implementation of federal policies. He went 
on to indicate that no school could be identified without missing AYP twice consecutively and, therefore, 
could not be identified in the absence of the new flexibility that was available for the second AYP year. 

Although not mentioned in North Carolina’s request for amendments or Senator Kennedy’s letter, 
provisions of Part D of Title IX of NCLB (section 9401) do appear to permit the Secretary to approve 
retroactive application of amendments to a state accountability plan through the law’s provisions for waivers 
of statutory and regulatory requirements. Retroactive application in this case is not included among the law’s 
restrictions on waivers (section 9401(c)) nor is it prohibited under section 1111(g)(1)(B) pertaining to the 
extension of deadlines for state assessment and accountability systems. 

In a very different approach to retroactive applications in AYP decision-making, Kentucky incorporated in 
its 2002-03 accountability determinations changes that were only later submitted to ED for its review. The 
changes were eventually approved and the state’s 2002-03 AYP decisions were allowed to stand with the 
following comment the approval letter from ED: 

I also understand that Kentucky made adequate yearly progress (AYP) determinations for the 2002- 
2003 school year using several of the amendments listed . . . that had been submitted to the 
Department, but not yet approved. This unacceptable action impedes the Secretary’s duty to ensure 
that States are implementing accountability plans that meet the requirements of Title I, Part A of 
NCLB. The Secretary takes this duty seriously and is inclined to withhold Title I funds from any 
State that does not fulfill its assurance of implementing its approved plan. By not using its approved 
plan to make AYP determinations for school year 2002-2003, Kentucky is subject to the loss of 
federal funding. In this one instance, however, the Secretary will not withhold Kentucky’s federal 
funds only because the criteria it used to make AYP determinations are compliant with NCLB. In the 
future, do not deviate from your amended plan as approved by the Department, lest you be subject to 
financial consequences (Simon, February 2, 2004). 

ED’s position with respect to retroactive application of approved changes to how a state makes AYP 
determination is somewhat perplexing. The law seems to permit just such an action, and ED decided not to 
penalize Kentucky for effectively making retroactive applications because the applications were “compliant 
with NCLB.” In almost all cases, states sought changes to their AYP models — and wanted to apply them 
retroactively — because ED’s related approval decisions changed in so many areas as the 2003 Peer Reviews 
progressed. Several AYP components originally denied were later approved. ED also made a number of 
“surprise” approvals and later described those as evidencing new flexibility around implementation of the 
law. Further, some states have challenged the part of ED’s rationale that making the amendments retroactive 
would result in no practical difference, and would also result in great confusion in the field, believing that 
these concerns are erroneous and best left to the states themselves to decide. 



7 Thirty other members of Congress co-signed this letter. Technically, additional regulations pertaining to LEP students 
have not been issued. The Secretary did issue a letter to states (February 20, 2004) setting forth additional flexibility 
under the law related to this subgroup. The Secretary issued two additional flexibility letters on March 2, 2004 (regard- 
ing SWDs) and March 29, 2004 (regarding participation rates). 
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AYP Indicators 

Most states requested amendments related to one or more of the five indicators required for calculating AYP. 
These include percent proficient for reading or language arts, percent proficient for mathematics, 
participation rate for reading or language arts, participation rate for mathematics, and at least one other 
academic indicator (OAI). Graduation rate must be included as an OAI at the high school level. 

Only one state appears to have made an amendment request that would apply to the calculation of all AYP 
indicators. Texas received approval to “apply standard rounding rules to all aspects of AYP elements 
including performance targets, performance rates, participation targets, and participation rates.” The state 
will round decimals of 0.5 or higher to the “next highest integer” and drop decimals of .4 or lower. 
Interestingly, a state reported that ED had informed it that the department woidd approve rounding rules in 
which all decimals are rounded up to the next whole number, e.g., a participation rate of 94.1 could be 
rounded to 95. 

Percent Proficient 

Very few states requested amendments related to the calculation of the percent proficient. It is worth noting 
that states were not required to provide the algorithm they use or even to describe this algorithm they use in 
calculating percent proficient in their accountability workbooks. Some states use all students enrolled for a 
full academic year (FAY) in the denominator and others use only those students who have been enrolled for a 
full academic year and actually took the test. In the June 2003 CCSSO paper on the initial accountability 
plans, we reported that Maryland and Georgia were the first states allowed to calculate percent proficient 
using the number tested in the denominator (Erpenbach, Forte Fast, & Potts, 2003, p. 15). In an un-circulated 
2003 Policy Brief, ED, commenting on the calculation of participation rate, noted that, “The State may, but is 
not required to, assign a score of ‘non-proficient’ for any students who do not participate in the assessment. 
This policy requires a State to count as a ‘participant’ only those students who actually complete some 
portion of the assessment.” In at least the Maryland and Georgia cases, this was interpreted to mean that 
students who do not participate in the assessments do not have to be considered non-proficient when 
calculating the percent proficient. This essentially removes these students from the denominator for that 
statistic. It is possible that other states may be calculating percent proficient with only tested students in the 
denominator but have not made this public. 

Most of the amendment requests in this area related to changes to their definition of full academic year. 

States are required to include in the calculation of percent proficient only scores for students who have 
attended the school for at least a full academic year. Oregon received approval for defining full academic 
year as enrollment for at least one-half of the instructional days prior to May 1 of each year. One other state 
would like to define full academic year as enrollment for at least 173 days in the current school year; two 
others would like to consider students to have been enrolled for a full academic year if they are enrolled at a 
specific point in the fall and also enrolled at the time of the test. ED approved (July 1, 2004) Kentucky’s 
new definition of a full academic year as enrollment for at least 1 00 instructional days counted from the first 
instructional day to the first day of the state testing window. 

Wyoming received approval to “use up to two years of achievement data when making AYP 
determinations.” The state will make a secondary examination based on averaged data whenever a school or 
district does not make AYP on the basis of current year results. Iowa received approval (June 17, 2004) to 
“clarify its definition of full academic year for students with IEPs who participate in the alternate 
assessment. Since the Iowa Alternate Assessment is a collection of evidence over time, the specifications 
need to indicate the length of time over which evidence is collected, and what happens to the evidence.” 
Effectively, then, full academic year will be defined for some SWDs as being more than one school or 
calendar year. 
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One other state has requested permission to use a growth model in lieu of percent proficient; this state did 
not describe the particular model it would like to use. 

Participation Rate 

Of all the changes states requested to their accountability designs, calculation of participation rate received 
far and away the greatest attention. Clearly, the decision ED Secretary Rod Paige announced on March 29, 
2004, to provide increased flexibility with regard to how states may calculate participation rate in making 
AYP determinations was a welcome change in an area proving to be troublesome for many states. 

While the number of schools missing solely the participation rate requirement since the enactment of NCLB 
is unknown, it appears to be considerable. Robelen (Education Week, 2004) profiled the experience of 
several states and school districts, concluding, “In many cases across the country, schools just narrowly 
missed the 95 percent participation rate last year.” In California, for example, nearly two-thirds of the high 
schools missed only this AYP requirement. Texas initially identified about 1 ,000 schools as not making AYP 
in 2002-03 with participation rate being the leading cause. Earlier, Pennsylvania, in handling hundreds of 
appeals from schools identified for improvement, decided to round up test-participation rates of 94.5 or 
above (Gewertz, 2003). 

Policies for Calculating Participation Rate (March 29, 2004) 

A state may use data from the previous one or two years to average participation 
rate determinations for a school or subgroup. 

Students unable to take tests because of a unique, significant medical emergency do 
not have to be counted in calculating participation rate. 

Originally, participation rate was to be calculated using the number of students enrolled at the time of testing 
as the denominator and the number of students who participated in the assessments in the numerator. This 
statistic was to be calculated independently for each school year. No enrolled students were to be excluded 
from the denominator. That is, prior to releasing this guidance, ED allowed no exemptions to testing and 
recommended that states handle exemptions due to medical emergencies as part of the appeals process under 
sec. 1116(b)(2) for schools and sec. 1116(c)(5) for LEAs. 

Each of the states that asked to apply the new flexibility to the AYP analyses for 2003-04 school year has 
received approval from ED — as long as the request fits within the parameters of the guidance. 

It should be noted that in December 2003, ED rejected West Virginia’s request to develop state criteria that 
would allow for limited exemptions of students “when an unexpected medical or psychological condition 
prohibits participation.” ED indicated that that the state could not establish “preemptive criteria for such 
exclusions” but may consider them on a case-by-case basis through the appeals process. 

As an example of how states have incorporated the new guidance into their plans, Ohio plans to use two-year 
weighted averages for calculating participation rates. In calculating “weighted” averages, the state will take 
into account the actual denominators and numerators for each year and will not simply average the 
participation rates from the two years. North Carolina is another state that will adopt multi-year averaging 
for this AYP requirement. 

Minimum “n ’’for Participation Rate Calculations 

Requests that were not related to the new guidance included one for an increase in the minimum “n” (see the 
“ Strategies Intended to Enhance Reliability ” section of this paper for a definition of minimum “n”) for 
participation rate from 20 to 40. Kentucky received approval (July 1, 2004) to revise its minimum “n” for 
participation rate from 10 students per grade and 30 students per school to 10 students per grade and 60 
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students per school or 15 percent of the school population. In other words, Kentucky will not calculate 
participation rates for any group with fewer than ten students per grade and 60 students per school unless 
that group represents at least 15 percent of the enrollment at the school. 

Two other states would like to adjust the participation rate target such that schools and subgroups with fewer 
than five students not tested, regardless of enrollment, would be considered to have met the participation rate 
requirements. A variety of strategies to reduce the impact of small numbers of non-tested students in small 
schools were included in states’ original accountability plans; these “fewer than five” not tested requests for 
amendment may provide a simpler option if approved. 

Students “Opted-Out” of Testing 

In some states, parents are allowed by statute or regulations to exclude their children from state testing 
programs. These “opt-out” provisions can mean that a school could miss the participation rate target even if 
every other student participated. To protect schools in these situations, one state requested approval to omit 
opted-out students from the denominator of the participation rate. Parents in this state may opt their children 
out of assessments, although only a few do so each year. ED denied the request verbally, but did not mention 
this request in its response letter, and has not subsequently provided a written rationale or explanation for its 
non-response. Instead, Secretary Paige may have been referring to this issue when he commented in the 
approval letter, “those changes that are aligned with NCLB are now included in an amended State 
accountability plan.” 

Another state would like to use a more complex strategy. If a group/school/district did not meet its percent 
proficient target, then it did not make AYP regardless of participation rate; however, if a student group, 
school, or district met the percent proficient target and tested at least 90 percent of the enrolled students, the 
number of students opted-out would be compared with the number needed to raise the participation rate to 95 
percent. If the number of opted-out students equals or exceeds the number needed to make 95 percent 
participation, then the participation rate target is considered met and the percent proficient is recalculated 
with the opted-out students counting as not proficient. The recalculated percent proficient is then compared 
to the target. This approach would seem to be inconsistent with ED’s policy on not counting as “non- 
proficient” students who do not participate in state assessments (see above under “Percent Proficient”). 

Statistical Tests Applied to Participation Rate Determinations 

The 2003 CCSSO report (Erpenbach, Forte Fast, & Potts, 2003) documented that ED’s perspective on the 
application of statistical tests for the purpose of enhancing the reliability of AYP determinations changed 
over time. Initially, ED indicated that statistical tests such as confidence intervals could not be used with 
“count” indicators 8 (i.e., participation rate, attendance rate, and graduation rate). They also could not be used 
in safe harbor 9 reviews (application of confidence intervals to safe harbor is described in a subsequent 
section of this paper.). Some state accountability designs that were approved late in the review cycle in 
2003, however, included the use of confidence intervals with “count” items. For example, North Dakota 
was approved for the use of a 95 percent confidence interval for participation rate calculations. Other states 
noted this approval, especially if they had proposed this method earlier or had been initially interested in it 
but somewhat dissuaded by ED’s initial decisions. 



8 “Count” indicators are rates — that is, the result of dividing the number of students in a group that meet certain criteria 
by the total number of students in the group. Percent proficient is also a rate, but apparently is considered by ED to be a 
different category of indicator. 

9 See Appendix B in Marion, S., White, C., Carlson, D., Erpenbach, W. J., Rabinowitz, S. & Sheinker, J. (2002). 

Making valid and reliable decisions in determining adequate yearly progress. Washington, DC: Council of Chief State 
School Officers (available at http://www.ccsso.org/content/pdfs/AYPpaper.pdf), for a review of the safe harbor 
provisions and their application. 
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On initial review, it appears that ED continues to deny the use of a statistical test with “count” elements. 
Both Tennessee’s request to use confidence intervals in attendance and graduation rate calculations and 
another state’s request to use them in participation rate calculations were denied. Many states, however, 
appear to have approval to use confidence intervals on such indicators; ED approved their proposed 
amendments to use confidence intervals for “AYP calculations” or “AYP decisions” without reference to 
(without exclusion of) specific AYP elements. 

Other Approaches to Calculating Participation Rate 

Kansas received approval for its request to use the number of test answer documents as the denominator in 
its participation rate calculation because the state does not currently have a student-level database. A number 
of other states already use this strategy; all are required to enforce the return of answer documents for every 
enrolled student, whether a student tests or not. Another state allows students to retest in the areas where 
they did not achieve a passing score on the first attempt and would like to base participation rates only on 
first-time test takers. 

Other Academic Indicators 

Requests for amendments to other academic indicators (OAIs) relate either to those used at the elementary 
and middle school levels or to graduation rate (required) at the high school level. 

Other Academic Indicators at the Elementary and Middle School Levels 

Several states requested changes to their other academic indicators at the elementary and middle school 
levels. The most common change seemed to involve a shift from requiring schools and districts to meet or 
exceed a specific target to one of making progress toward the target (Hawaii being an exception). For 
example, Tennessee changed its AYP requirement that schools and districts meet or exceed an attendance 
rate of 93 percent at the elementary school level and a graduation rate of 90 percent at the high school level 
to also allow for any degree of progress over rates from the previous year. This state also requested approval 
to employ a 95 percent confidence interval with these rates, which ED denied. Schools and districts above 
the targets are not required to make further progress. 

Pennsylvania changed its AYP targets for attendance and graduation rate from 95 percent each to 90 percent 
and 80 percent, respectively, to permit continued progress toward these targets for schools and districts 
below them. Kansas dropped an attendance rate requirement at the high school level that it had originally 
applied for AYP determinations in addition to the graduation rate requirement at this level. Other states have 
requested a simple change to the status target or the use of a single year or the average of three years of data, 
whichever provides the greater benefit to the school. 

Several other states requested permission to replace the indicator described in their original accountability 
plans with another indicator. Hawaii will use a retention rate for its OAI (the percent of repeating 8 th grade 
students) but will require that schools meet or exceed the state-set targets; not just make improvement over 
the previous year. Virginia will now allow elementary and middle schools to choose either the current OAI, 
attendance with a target of 94 percent, or performance on the state’s science assessment with a target of 70- 
percent proficient. Kansas will be allowed to remove attendance as an OAI for high schools. 

Kentucky will be allowed to lag its OAIs by one year in order to ensure reliability of the data used. It will 
also be able to use its accountability index rather than its academic index for the OAI. The academic index 
includes only achievement information while the accountability index includes attendance and retention in 
the elementary school grades and attendance, retention, and school drop-outs in the middle grades. 
Graduation rate will remain as the only OAI used at the high school level. 
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